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ABSTRACT 

Three influences on the outcomes of literature 
searches undertaken as part of integrative research reviews were 
examined: (1) the degree of expertise of the searcher; (2) the amount 
of information available (i.e., keywords, bibliographies, abstracts); 
and (3) the cognitive characteristics of the searcher. Participants 
were presented with descriptions of documents varying in topic area 
(related or not related to their expertise), amount of information, 
and whether the document was relevant or irrelevant to the search. 
Results revealed no difference between experts and nonexperts in 
their accuracy of judging relevant documents but experts were better 
able to discern that irrelevant documents were, in fact, irrelevant. 
Judgments based on abstracts were more accurate than judgments based 
on keywords or bibliographies, which did not differ, and this effect 
was more pronounced for relevant than irrelevant articles. 
Pa u^°u P j nts who had P revi °wsly conducted more literature searches or 
published more research reviews made more accurate judgments as did 
searchers high in cognitive complexity or tolerance of ambiguity. 
Data are also presented on how the independent variables related to 
confidence in relevance judgments, familiarity with the literature, 
and general beliefs about topic areas. Results are discussed in 
regard to prevailing beliefs about the literature search process and 
how the process might bo improved. (Author/THC) 
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Abstract 

Three influences on the outcomes of literature searches 
undertaken as part of integrative research reviews were examined: 
(a) the degree of expertise of the searcher; (b) the amount of 
information available (i.e. keywords* bibliographies, abstracts); 
and (c) the cognitive chacteristics of the searcher. Participants 
were presented with descriptions of documents varying in (a) topic 
area (related or not related to their expertise), (b) amount of 
information and (c) whether the document was relevant or 
irrelevant to the search. Results revealed no difference between 
experts and nonexperts in their accuracy of juuging relevant 
documents but experts were better able to discern that irrelevant 
documents were, in fact, irrelevant. Judgments based on abstracts 
were more accurate than judgments based on keywords or 
bibliographies, which did not differ, and this effect was more 
pronounced for relevant than Irrelevant articles. Participants 
who had previously conducted more literature searches or published 
more research reviews made more accurate judgments. Searchers 
high In cognitive complexity or tolerance of ambiguity made more 
accurate judgments. Data was also presented on how the 
independent variables related to: (a) confidence in relevance 
judgments; (b) familiarity with the literature, and; (c) general 
beliefs about topic areas. Results are discussed in regard to 
prevailing beliefs about the literature search process and how the 
process might be Improved. 
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Influences on the Outcomes of Literature Searches 
for Integrative Research Reviews 

Since the publication of Smith and Glass' (1977) meta-analysis 
on the effects of psychotherapy, psychologists have taken a keen 
Interest In quantitative procedures for summarizing and Integrating 
empirical research findings. Proponents and opponents have argued 
over whether employing statistical procedures In research reviews 
makes their outcomes more or less valid. 

Regardless of how one views meta-analysis, a positive 
consequence of the surrounding debate 1s that it has Increased the 
attention paid to other aspects on integrative research reviewing 
(Cooper, 1964). For Instance, research reviewers are now expected 
to take greater care 1n conducting their literature searches and 
in reporting their searc . procedures. Psychologists are realizing 
that the validity of a review's conclusions is as much a function 
of the legitimacy of the searching procedures as it Is of the 
Integrating process. 

This paper reports a study examining some of the factors that 
Influence literature searches conducted as part of integrative 
research reviews. While information scientists have closely 
scrutinized certain aspects of the literature searching process, 
psychologists are largely unaware of these efforts and their 
Implications for the types of reviews they conduct. In addition, 
crucial aspects of searches performed for integrative research 
reviews have largely been ignored by both disciplines. 

The present study focused on four aspects of the literature 
searching process — the amount of information a reviewer has when 
making searching decisions, and the topic expertise, search- ng 
experience, and cognitive characteristics of the searcher. 

Amount of Information 

The Information about documents a searcher Initially 
encounters comes 1n three forms: keyword*, bibliographic 
information, and abstract*. Keywords are critical terms meant to 
describe the contents of the documents. These may be supplied by 
the author, attached by Indexers employed by the literature 
database service (using a thesaurus of accepted Index terms), or 
located 1n titles and abstracts by computer scanning. 

Intuitively, the accuracy of a relevance decision should be 
greatly Influenced by the amount of available information. In two 
previous studies, the effect of Information on relevance decisions 
was examined. These studies defined as "relevant documents" those 
judged applicable to a search based on a reading of the entire 
document. Rath, Resnlck, and Savage (1961) found little difference 
1n relevance Judgments based on sentences chosen systematically 
from documents (e.g. the first and last five percent of sentences) 
and on readings of the full document. They did find, hewever, th*t 
agreement between full document readings and titles only was low. 
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On the other hand, Dym (1967) found that the probability of a 
judgment based on bibliographic Information corresponding exactly 
with one based on the entire text was .69. The addition of an 
abstract to the bibliographic Information only raised the 
probability to .73. No previous research has compared the utility 
of keywords to that of titles or abstracts. 

In the present study, all three levels of information 
encountered In the course of performing a search were examined. 
Whether a search Is manual or computerized, the task begins with 
the specification of keywords and an examination of the longer 
lists of keywords attached to articles. Next, during a compu'er 
search, searchers can examine only the bibliographic information 
(authors, title, publication source) of documents, or they may 
retrieve the abstract along with the bibliographic Information. 
In manual searches, the abstract and bibliographic Information are 
typically retrieved together, after keywords have led the searcher 
to believe the document may be relevant. In searches employing 
techniques other than literature databases (e.g. scanning 
journals), bibliographic Information Is likely to be the first type 
of Information encountered. 

The keywords, bibliographic Information, and abstracts used in 
this study were all actual contents of literature databases, were 
associated with empirical and nonemplrlcal journal articles, and 
had been located as part *f prevl ous ly publ 1 shed Integra 1 1 ve 
research reviews. 

Topic Expertise 

The study of how expertise affects decision making has 
stimulated considerable psychological research. For Instance, 
research 1n the medical domain often focuses on how exoerts and 
novices formulate diagnostic questions, how the nature of these 
questions may differ, and how the resulting Information Is 
utilized or combined (£1 stein, Shulman, and Sprafka, 1978). 

Despite their surface dissimilarities, medical diagnosis 1s in 
many ways analogous to decision-making during a literature search. 
Searchers formulate inquiries (e.g. decide on a topic's breadth) 
and (..loose the channels of scientific communication through which 
they will gather Information (e.g. research literature databases 
and Index terms to search, journals to scan, colleagues to 
contact). It 1s more difficult 1n the context of literature 
searching than medical diagnosis, however, to assess the "utility" 
or "accuracy" of decisions during these Initial phases. For 
example, Cooper (1985) found that experts In a topic area differed 
from nonexperts not only 1n hot they conducted a literature search 
but aUo 1n how they construed the tasks — experts were less 
Interested In writing research reviews that exhaustively covered a 
literature. In medical diagnosis the task 1s more Invariant. 

The outcome of the Initial phase of a literature search 1s a 
series of incomplete descriptions (i.e., keywords or index terms, 
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bibliographic Information, abstracts) of potentially relevant 
pieces of Information (I.e., journal articles or other documents). 
From these descriptions the searcher makes numerous discrete 
decisions about whether to pursue the Individual documents further. 
Nedlcal experts might be confronted with a similar series of 
decisions. However, medical decision-makers may follow different 
paths, of Indetermlnant accuracy, to arrive at a similar overall 
diagnosis. The literature searcher's decision about each piece of 
Information, 1n contrast, can be assessed for Its Individual 
accuracy, that Is, whether It leads to a document containing 
Information relevant to the research review. It Is this second 
phase o# the literature search, Involving decisions about the 
relevance of documents based on Incomplete Information, that Is the 
focus of this study. 

Research in the Information sciences Indicates that judges 
with greater knowledge of a topic area may show greater agreement 
among themselves concerning whether particular articles are 
relevant to a search. For example, Cuadra and Katter (1967) 
divided senior psychology majors and first year graduate students 
Into four levels of expertise, based on their number of academic 
credits 1n psychology and their knowledge of psychology authors. 
Students then judged the relevance of nine journal abstracts for 
answering eight questions. The average correlation of judgments 
withln-groups Indicated that Individuals 1n the more knowledgeable 
groups agreed more often about the relevance of the abstracts to 
the search. However, for all four groups the average correlations 
varied only between .41 and .49. 

In another study, Rees and Schultz (1967) employed a similar 
research design but judges varied 1n medical, rather then 
psychological, expertise. They found that not only did medical 
scientists, the group with the greatest expertise, show the 
greatest wlthln-group agreement but this group also judged the 
fewest number of documents as relevant to the search. 

Research examining the effects of topic expertise on 
literature searching has largely been abandoned for the past two 
decades. Two reasons account for this development. One Involves 
problems associated with defining the term "relevance", and by 
Implication "judgmental accuracy", 1n a manner that 1s meaningful 
across all the forms of Inquiry encountered by Information 
scientists (SaracevK, 1970, 1975). Thus, In both studies 
mentioned above, Inter judge agreement was scrutinized 1n Heu of 
Judgment accuracy. 

In the present study a satisfactory operational definition of 
accuracy could be realized because of the unique nature of the 
Integrative research reviewing task. The accuracy of judgments was 
made measurable by using as stimuli documents known, a priori » to 
be either relevant or Irrelevant to the searchers' task. 
Specifically, searchers were asked to estimate the likelihood that 
documents contained empirical data relevant to the topic of an 
Integrative research review. Whether the documents did in fact 



Literature Searches 6 



contain such data was manipulated as an independent variable. 
Therefore, "greater accuracy" In this study describes a finding 
when documents that actually contained empirical data were assigned 
a greater likelihood of containing such data and/or when documents 
without empirical data were assigned lesser likelihoods. "Lesser 
accuracy" describes the "oposlte situations. 

The second reason for the abandonment of topic expertise 
studies was that the advent of computerized literature databases 
created a whole new area requiring critical Investigation. One 
question given added Importance by the computerization of 
literatures was whether experience with this searching mechanism 
leads to more satisfactory search outcomes. 

Search Experience 

How past experience affects the searching process has recently 
been a much studied question In information science (Fenichel, 
1980; Howard, 1982; Oldroyd, 1984). Host studies demonstrate that 
experienced searchers perform more precise and cost-effective 
searches than Inexperienced searchers. According to Bates (1977) 
search experience leads to a familiarity with subject headings and 
how they are applied. Hc/ever, these studies are conflnid to the 
initial phases of searching, that is, to the definition of queries, 
the choice of databases to search, and the selection of keywords. 
No studies have examined the effects of searching experience on the 
searchers ability to distinguish relevant from irrelevant documents 
based on incomplete Information. In addition, past research has 
used as seirchers librarians or library science students, typically 
with different levels of computer searching experience or varying 
degrees of exposure to particular literature databases (e.g. ERIC, 
Psychlnfo). 

The present study examined the effect of searching experience 
on the accuracy of relevance decisions based on keywords, 
bibliographic information, and abstracts. The searchers were 
"end-users", that 1s, the Individuals who Intend to use the outcome 
of the search to produce new scholarly documents. In addition to 
search experience, this study also examined whether past experience 
at writing and publishing Integrative research reviews also 
affected relevance decisions. 

Cognitive Characteristics 

While it could be expected that the cognitive chai <*Ueristics 
of searchers would Influence decisions concerning the relevance of 
documents, surprisingly few studies have examined this Issue. 
Davidson (1977) noted that while "Individuals selecting 
Information from alternatives do so in a somewhat Idiosyncratic 
fashion (existing library systems deal) with users as though they 
were all the same" (p. 273). 

The present study examined two cognitive characteristics of 
searchers: tolerance of ambiguity and cognitive complexity. 



7 



Literature Searches 7 



Budner (1962) defined Intolerance of ambiguity as "the tendency to 
perceive (i.e. Interpret" J ambiguous situations as undesirable" (p. 
29). Allport (1954) Maintained that Intolerance of ambiguity 1s 
Manifested through 3 need for concreteness and definlteness. 
Persons who are Intolerant of anblgulty would be more 
uncomfortable with unclear or unfamiliar situations. 
Frenkel-Brunswlk (1949) held that people Intolerant of ambiguity 
value a clear perception more than a correct one. 

With regard to cognitive complexity, persons who are low on 
this dimension attempt to reduce everything to a few simple 
components (Honess, 1976). In contrast, cognitive ly complex 
Individuals possess more cognitive categories In more complex 
interrelation. They are able to perceive differences In a more 
sophisticated manner, and to better Integrate Information, both 
positive and negative, Into a coherent whole (Campbell, 1960; 
B1er1. 1968). 

Hypotheses 

In roncluslon, the primary hypotheses of this study, based on 
the above research and, when research was not available, on 
experience and some Intuition, were: 

(a) Judgments of the relevance of an article, when searching 
for empirical studies to Include 1n an Integrative research 
review, will be more accurate 1f they are based on bibliographic 
Information and an abstract than on bibliographic Information 
alone. Likewise, bibliographic information will lead to more 
accurate judgments than the keywords used to Index the articles. 
Confidence in judgments of relevance will follow a similar 
pattern. 

(b) Experts will make more accurate judgments than nonexperts 
of the relevance of articles 1n their topic area. Experts will 
also be more confident about their judgments. 

(c) Persons with more experience at performing literature 
searches and/or at publishing research reviews will be more 
accurate judges of i>e relevance of articles to their search. 

(d) Persons high 1n cognitive complexity and/or tolerance of 
ambiguity will generally judge articles as more relevant to their 
search than will persons low 1n these characteristics. 

Methods 

Ov erflew of Design 

The basic study design Involved the manipulation of three 
Independent variables associated with performing a literature 
search: (a) the topic area of studies, el t ar the nation of 
locus of control to academic achievement or ckt effects of drug 
treatarcnts on hyperactivity; (b) the amount of Information about 
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an article given to participants, either keywords only, 
bibliographic citation, or bibliographic citation and abstract, 
and; (c) the article's content, that this, whether or not the 
article w*s relevant to the participants' searching goal. All 
three Independent variables w* r « manipulated wi thin- subjects, 
constitution a 2x3x2 repeated measures design. 

The major dependent variables were the participants': (a) 
estimate of the likelihood that a given article was relevant to 
their search, and; (b) confidence In their likelihood estimates. 
In addition, measures of the participants' familiarity with the 
literature and general beliefs about the two topic area were 
collected. 

Several Individual difference measures were also collected to 
determine If they moderated the effects of the manipulations. 
These Included: (a) the expertise of the participants in the two 
topic areas; (b) the number of Integrative research reviews the 
participants had published; (c) the nuar.ber of literature searches 
the participants had conducted; (d) the participants' cognitive 
complexity, and; (e) the participants' tolerance of ambiguity. 
Each of these Individual differences was added to the basic 
research design as predictors of responses. 

Literature Search 

Both the selection of participants and the stimulus materials 
used In this study were facilitated by two literature searches 
conducted 1n connection with previously published Integrative 
research reviews. The first review, entitled "Locus of control 
and academic achievement: A literature review," was published by 
Flndley and Cooper (1983). The Initial bibliography for this 
review was constructed through an online computer search of 
Psychological Abstracts . Resources In Education , and Dissertation 
Abstracts international . The keywords used in the searcn were 
crossings of the terms achievement or performance with locus of 
control or Internal "external . The three databases contained 802 
studies that had both of these terms In their title or abstract. 
A reading of the abstracts led the initial reviewers to examine 
208 articles in their entirety. Of the 208, 98 articles were 
deemed relevant to the goal of the review. To be considered 
relevant, a study had to Include (a) a measure of lecus of 
control, (b) a measure of academic achievement, and (c) a test of 
the relation between the two measures. 

The second review, entitled "Drug treatment of hyperactivity 
In children," was conducted by Ottenbacher and Cooper (1983). 
Stuoles potentially relevant to this topic were obtained through 
computer searches of Index Hedlcus . Psychological Abstracts . 
Current Index to Journals in Education . Exceptional Children 
Educational Resources , and Dissertation Abstracts I nternational . 
A tot*) of 831 study a bs t ra cts were retrieved. oT\hese studies , 
61 me: the relevance criteria of (a) Investigating the effect of a 
drug treatment on children diagnosed as hyperactive; (b) using a 
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two group comparison Including some form of control group; (c) 
employing random ass lament and a double blind procedure as part 
of the design, and; (d) reporting the results In a fashion that 
allowed quantitative analysis. 

In the discussion that follows, the term "relevant" Is used 
to describe articles that were chosen for Inclusion In either of 
the two literature reviews because they contained empirical data 
relating to the direction and size of the relation being reviewed, 
that 1s between locus of control and academic achievement or the 
effect of drug treatment on hyperactivity. The term "Irrelevant" 
1s used to describe an article not Included In either of the two 
reviews and which contained no analyzable empirical data. These 
articles, of course, could be pertinent to the topics In ways not 
related to the present study or to the judgment the participants 
were asked to make. The Irrelevant articles were chosen from 
articles uncovered by the literature searches but not used in the 
research reviews. 

Participants 

Participants for the study were recruited through mail 
solicitation. Of the 98 first authors of locus of control and 
academic achievement studies, 67 were Invited to take part In the 
study. Of the 61 hyperactivity first authors, 50 were Invited to 
take part. Invitations were based on the availability of the 
author's address. Authors were paid a $25 honorarium for time 
spent on the project. 

For the locus of control sample, 2* of the 61 Invitees did 
not respond, 19 declined the Invitation, 7 accepted the Invitation 
but never returned the materials, and 17 (255J returned completed 
questionnaires. For the hyperactivity sample, 18 of the 50 
Invitees did not respond, 15 declined the Invitation, 3 accepted 
the Invitation but did not return the materials, and 14 (281) 
returned completed questionnaires. 

?n sum, 31 authors of empirical articles participated In the 
study, with 17 having topical expertise 1n locus of control and 
academic achievement and 14 In drug treatment of hyperactivity. 
Topical expertise , then, was Initially defined as a participant 
having published one article relevant to one of the topics. The 
expertise labelling of participants was later cross-validated by 
responses to a section of the questionnaire which asked authors to 
tally their publications and to rank their "standing among 
scholars" In both fields. A six point scale, ranging from 1, "w 
person new to the area", to 6, "a leader In the topic area" was 
used. Only two authors claimed to have made published 
contributions to the topic area to which they hao not been 
assigned (both were hyperactivity experts claiming contributions 
to the locus of control and achievement literature) and no 
participant claimed a standing among scholars In their nonexpert 
area within two scale points of their expert area. 
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In general, hyperactivity experts claimed somewhat higher 
standing 1n their field than did locus of control experts. 
Hyperactivity experts averaged a rating on the "standing among 
scholars" scale of 5, equivalent to "an active, continuing 
contributor". Locus of control experts average 3.8, wit* 10 
authors call Ing themselves "a former contributor, no longer 
active" and 6 calling themselves "a person who made a single 
contribution". 

Stimulus Materials 

The stimulus materials were presented to participants in two 
separate sections, one focusing on locus of control and one on 
hypera :*<vity. Each section began with a background sheet which 
briefly Introduced the substantive topic un*er scrutiny and the 
nature of the task. 

The locus cf control and academic achievement section began 
with the following description of the topic: 

Locus of control refers to people's feelings about whether 
they control the events that occur In their life. Sorce people, 
labeled Internals, feel personally responsible for the things that 
happen to them. Other people, labeled externals, feel that their 
outcomes In life are determined by forces beyond their control, 
for example, fate ur other people. Locus of control beliefs are 
not so discrete, however— people form a continuum ranging from 
highly Internal belief systems to highly external ones, with most 
belief systems falling somewhere In between. 

One topic that has Intrigued both psychologists and educators 
Is the relationship between locus of control beliefs and 
achievement In school. The prediction that a stronger belief in 
Internal locus of control will be as<oc1ated with higher academic 
achievement Is both logical and consistent with relevant theories. 
People who feel more able to control the outcome of events should 
exert more effort to do so. Also, Internals should experience 
more pride when they succeed and more shame when they fall. Thus* 
the rewards and punishments experienced in school by Internals 
should be stronger and. therefore, more motivating than those 
experienced by externals. 

The drug treatment of hyperactivity section began with the 
following description of the topic: 

Hyperactivity, a disorder Involving a deficit in attention, 
has been estimated to afflict between 41 and 201 of school -aged 
children. Out to its frequent occurrence, both educators and 
physicians have vigorously pursued effective treatments for 
hyperactivity. The proposed treatments range from dietary 
restrictions and Supplements to behavior modification training. 
By far the most prevalent treatment of hyperactivity, however, 1s 
pharmacological management and almost all children Identified as 
hyperactive receive some form of drug therapy 1n the course of 
their treatment. 
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In spite of the fact that drug treatment of hyperactivity is 
so frequently prescribed and evidence for Its effectiveness Is 
widespread, there Is still substantial controversy over the 
advisability of Its use (e.g., Its ethlcality and Its potential 
susceptibility to placebo effects). In addition, there has been 
some question about relative effectiveness of different types 
of drugs (stimulants and nonstlmulants) and about any drug's 
ability to affect behaviors beyond simple motor or perceptual 
performance (e.g., social adjustment and educational performance). 

After each topic description, the nature of the participants' 
task was described as follows (the changes In wording associated 
with the two topics appears In brackets): 

Assume you have decided to perform a literature review on the 
topic of [locus of control and academic achievement/drug treatment 
of hyperactivity]. The review will have the following goals* In 
descending order of Importance: (a) to describe , summarize and 
Interpret the findings of empirical research meant to uncover the 
direction and magnitude of the [locus of control and academic 
achievement link/drug treatment effect on hyperactivity]; (b) to 
Identify Important Issues associated with the topic (e.g., 
methodological and conceptual Issues, Interacting variables); and 
(c) to suggest topics for future research. 

Your goal Is to uncover as much literature on the relation as 
you possibly can. This does not mean, of course, that every 
relevant work will be cited In your paper. However, before you 
begin the summarizing and Interpreting process, you want your 
personal file of relevant works to be as exhaustive as possible. 
To help you construct this file, you ran computerized literature 
searches of the [Psych Info and ERIC/Index Medlcus, Psychological 
Abstracts, ERIC and Exceptional Children Education Resources] 
abstracting services, using the keyword(s) ["achievement" or 
"performance" crossed with "locus of control" or 
"1nUmaMxtertu<r/"hyperact1v1ty*]. The searches yielded a 
total of (802/831) citations, so It 1s Impractical for you to 
examine all of the full reports. Assume also that the abstract 
service Information that follows represents a random sample from 
the full list {and only the beginning of a much longer task). 

Finally, assume you Intend to submit the resulting review 
paper for publication In a scholarly journal like Psychological 
Bulletin or [the Review of Educational Research/Developmental 
Medicine and Child Neuroiogy' T Thus, your primary audience is 
otner researchers interested In the topic. 

On the pages that follow, you will find abstracting service 
Information on 30 articles uncovered by your computer search. For 
10 articles only the keywords or Index terms associated with the 
manuscript are given, for 10 articles the bibliographic reference 
Is given, and for 10 the reference and abstract are given. 
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The order In which the three levels of Information are 
presented corresponds with how Information Is typically 
encountered In actual searches. That Is, first you are asked to 
make judgments based only on keywords, then judgments based on 
bibliographic references and then on abstracts. 

Below each entry you are asked several questions concerning 
whether or not you anticipate the article contains relevant 
Information. Please complete these questions In as 
realistic a manner as possible. Complete them In the order In 
which they are presented. 

After reading the locus of control and academic achlevemeit 
topic and task description, participants were presented with 
thirty pages, each of which contained Information about a single 
locus of control article. The thirty article descriptions varied 
In (a) the amount of Information about the article that was 
presented an3 (bj whether or not the article described was 
relevant or Irrelevant to the task (called article content ). 

As stated In the Instructions, the first ten pages contained 
only the keywords (I.e. Index terms used by the computer) 
associated with each of the ten articles. The second ten pages 
contained the bibliographic citation (I.e. author(s), year of 
publication, title, journal, volume, and page numbers) associated 
with each article. The final ten pages contained the 
bibliographic citation and abstract associated with each article. 

Within each amount of information condition, five of the ten 
articles were preselected so as to be relevant to the task of the 
participant (I.e. to contain empirical information about the locus 
of control and academic achievement relationship). The other five 
articles contained only Irrelevant Information (to the empirical 
relation, though other t;'pes of Important Insights way have been 
contained In these articles). Relevant and Irrelevant articles 
were alternated within each condition. Of course, participants 
fat<-e not Informed that article content was being manipulated. 

All subjects also completed a section containing a topic and 
task background sheet and thirty articles concerning drug 
treatment of hyperactivity. The section on hyperactivity articles 
were constructed In a manner Identical to the locus of control 
section. 

The thirty articles In each topic section were chosen from a 
pool of sixty articles (thirty relevant and thirty Irrelevant) and 
each article appeared an equal number of times In the study. No 
article appeared twice, that Is In more than one amount of 
Information condition, In the materials for a single participant, 
though articles appeared In all conditions across participants. 
Half of the participants completed the locus of control section 
first and half completed the hyperactivity section first, without 
regard to the participants 1 topical expertise. Precautions were 
also taken to ninlnize the chance that participants would be asked 
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to make judgments about studies they had personally authored, 
although this did occur twice In the 1860 article ratings. 

Individual Olffcrcncc Measures 

Publication and Literature Searching Background , The number 
of research reviews participants had published was measured by 
asking them If they had every published an integrative research 
review and, If so, how many. On average, participants claimed 
3.83 published reviews f with a range from 0 to 31 and a median of 
2. For use as an Individual difference variable , labelled review 
publications , participants were divided, using an approximate 
median split, Into those with less than two published research 
reviews and those with two or more published reviews. 

Participants were also asked how many computerized literature 
searches they had conducted (or had conducted for them) and how 
many manual searches of an abstracting service they had performed. 
Participants claimed an average of 6.80 (sd«l0.06) computerized 
searches and 8.23 (sd* 14.06) manual searches. The number of 
computerized and manual searches were summed for each participant 
In order to form 4 single search experience variable. 
Participants were then divided, using a median split. Into those 
who had conducted less than sixteen searches and those who had 
conducted sixteen or more searches. 

Cognitive Complexity . A revised version of the Paragraph 
Completion Test (PCT; Schroder and Streufort, 1962) was employed 
to measure cognitive complexity. Participants were presented five 
sets of words or phrases that began an Incomplete sentence and 
were asked to complete the sentence and write at least one 
additional sentence following from the first. The five Incomplete 
sentences were, "When scholars disagree with ^ne another . . . ", 
"When t am In doubt . . . ", "Rules . . . ", "When others 
criticize me It usually means . . . ", and "Confusion . . . ". 

Scoring was accomplished In accordance with Schroder's (1971) 
seven-point scaling method. For a score of 1, representing low 
complexity, the response must be generated by a single fixed rule, 
with a high degree of certainty. A score of 3 (medium complexity) 
Is given if the response clearly contains reference to the 
availability of alternate means of perceiving the event. A score 
of 5 (medium-high complexity) Indicates that in addition to the 
consideration of alternatives the respondent proposes some kind of 
joint outcome. Finally, high complexity (a score of 7) means that 
the response reveals the functional relationships in the person's 
"world view". 

Each response was scored by two Independent judges. 
Inter judge reliability for the sum of the five responses was 
r*.43, similar to reliabilities found In other studies (see Tom, 
Cooper, and McGraw, 1984). The average of the two judges ratings 
was used as the measure of a participants* cognitive complexity. 
The mean total score for the sample was 17.30 (sd*3.96), or 3.46 
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per Item. The cognitive complexity variable was created by 
performing a median split dividing the sample Into high complexity 
and low complexity groups based on total scale scores. 

Tolerance of Ambiguity . Budner's Scale of Tolerance- 
Intolerance of Ambiguity I SUA; Budner, 1962) was used. The STIA 
contains sixteen Llkert-type Items accompanied by five-point 
scales ranging from "strongly agree" to "strongly disagree". This 
Instrument has shown acceptable intercorrelations with other 
measures of tolerance of ambiguity (r«.46; see Robinson and 
Shaver, 1973). 

Only eight of the items on the STIA were used in this study. 
An example of an Item used reads, "It is more fun to tackle a 
complicated problem than it Is to solve a simple one". Items were 
counterbalanced so that agreement could reflect both tolerance or 
Intolerance of ambiguity. 

Individual item responses were summed to create a single 
tolerance of ambiguity score, with higher scores denoting greater 
tolerance. The average score in the sample was 22.87 (sd*3.84). 
As with the other Individual difference variables, a median split 
was used to create high and low tolerance of ambiguity groups. 

Vannoy's (1965) study of 18 cognitive complexity measures 
demonstrated an Intercorrelatlon between the PCT and STIA of 
r*-*06. Tom, et.al. (1984) found r«.08. In the present study, 
the Intercorrelatlon was r».21. 

Dependent Variables 

The two primary dependent variables were the participants' 
(a) assessment of the likelihood that an article contained 
empirical data relevant to their search, and (b) confidence in the 
likelihood estimate. 

For locus of control and academic achievement studies, the 
question assessing the likelihood of containing empirical data was 
worded, "How likely is It that this paper contains empirical data 
(e.g., group comparisons, correlations) relevant to the direction 
and size of the relation between locus of control and academic 
achievement?" For drug treatment of hyperactivity studies, this 
question was worded Identically, except that "the effect of drug 
treatment of hyperactivity" was substituted for the reference to 
locus of control. Participants responded on an eleven-point 
scale, with 0 labelled "Not likely at all" and 10 labelled "very 
likely". 

For both locus of control and hyperactivity studies, the 
question assessing confidence in the likelihood estimate was 
worded, "How confident are you of the above estimate?" 
Participants again responded on an eleven-point scale, with 0 
labelled "not confident at all" and 10 labelled "very confident". 
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In the (a) bibliographic and (b) bibliographic and abstract 
information conditions* participants were also asked "Are you 
familiar with the content of this article". A "yes" or "no" 
response was requested. This question was omitted from the 
keyword Information condition since It was extremely unlikely that 
a particular article could be recognized by participants based 
solely on Its associated Index terms. 

Ancillary measures . In addition to the primary dependent 
variables* participants were asked several questions concerning 
their general beliefs about the two topic areas. 

Participants were asked to estimate (a> the average size of 
the relation between locus of control and academic achievement and 
(b) the average effect of drug treatment on hyperactivity. 
Participants expressed the locus of control and academic 
achievement relation 1n terms of the average correlation 
coefficient to be expected In a study. The drug treatment effect 
was expressed 1n terms of the IL-lndex, where U~ equals the 
percent of children In the dmg-treated group wno are less 
hyperactive than the median child 1n the untreated group (see 
Cohen, 1977). Participants were also asked to rate their 
confidence 1n their estimate, using the same confidence scale 
described above. 

Participants responded to three questions meant to measure 
more contextual beliefs about each topic area. Tnese questions* 
presented as attitude statements, were worded as follows (the 
changes In wording associated with the two topics appear In 
bracket:): (a) The link between [locus of control and academic 
achlevmnt/dmg treatment and hyperactivity] 1s highly complex, 
Involving many variables that mediate the relationship; (b) In 
general, problems In measurement mean that the outcomes of studies 
relating [locus of control and academic achievement/drug 
treatments and hyperactivity] are not reflective of the link as It 
exists In nature, and; (c) The [relation between locus of control 
and academic achievement/Impact of drua treatment on 
hyperactivity] Is an Important topic for [psychologists and 
educators/parents and the medical community]. All questions were 
responded to on an eleven-point scale* with -5 labelled "strongly 
disagree" and 5 labelled "strongly agree". 

Analytic Design 

Since participants responded to a total of sixty articles 
representing twelve distinct crossings of the three 
wlthln-subjects variables (two topics x three amounts of 
Information x two types of article content) the basic units of 
analysis were the average responses to the five articles that 
appeared In each condition. If any of the three wlthln-subjects 
factors was omitted from a particular analysis, responses were 
also aggregated over the omitted factor. Newman-Keuls post hoc 
means tests were employed where appropriate (see Myers* 1972). 
Degrees of freedom sometimes varied from one analysis to another 
because of missing data. 
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All analyses conducted on the data used the Analysis of 
Variance program from the Statistical Analysis System computer 
package (SAS, 1985). 

Results 

To pic Area, Topic Expertise* Amount of Information, and Article 
Content 

A series of ANOVAs containing one between-subject factor 
(topic expertise) and three within-subject factors (topic area, 
amount of Information, and article content) were performed on the 
three primary dependent variables (estimates of the likelihood 
that articles contained empirical data, confidence In the 
estimates, and familiarity with articles). 

LI ke 1 In ood es t Ima te s . The ANOVA performed on estimates of 
the likel<hood that an article contained date relevant to the 
search revealed numerous significant effects. The three-way 
Interaction, Involving topic area, topic expertise, and article 
content, underlled several of these effects. The associated means 
are displayed In Table 1. 

A significant main effect for topic area revealed that 
hyperactivity articles (H 3 6.81) were more likely to be judged as 
containing empirical data than were locus of control articles 
(M-6.22; F(l,29)-10.17» p .0034). A main effect for article 
content revealed that empirical articles (M*7.50) were judged more 
likely to contain empirical data than nonemplrlcal articles 
(M-5.55; F(l,29)-107.88, p .0001). A topic area by topic 
expertise Interaction revetled experts were less likely to believe 
articles In their area contained empirical data than article in 
their nonexpert area (F(l,29)«4.34, p .0461). 

A three-way Interaction (see Table 1) revealed that expt.-tise 
had little effect on estimates of the likelihood that empirical 
data was contained in articles that actually did contain empirical 
data. However, expertise did effect estimates for nonemplrlcal 
articles. Specifically* compared to nonexperts, experts judged 
nonemplrlcal articles in their area as being less likely to 
contain empirical data (F(1.29)"23.68, p .0001). 



Place Table 1 About Here 



Two significant effects Involved the amount of Information 
variable. A main effect revealed that article abstracts led to 
higher estimates that an article contained empirical data (M-6.94) 
than did bibliographic Information (M-6.^2) or keywords (M-6.28; 
F(2,58«)5.26» p .008). An Interaction between article content and 
amount of Information Indicated that abstracts Increased 
participants' estimates that empirical data was contained in 
articles that were in fact empirical but did not affect estimates 
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for nonemplrlcal articles (F(2,58)«9.96 p .0002). Table 2 
presents the means underlying this Interaction. 



Place Table 2 About Here 



Confidence In Judgments . The ANOVA conduc ted on the 
participants' ratings of confidence In their likelihood estimates 
also revealed several significant effects. Two main effects 
revealed that participants were more confident about their 
judgments concerning hyperactivity articles (N-7.81) than they 
were about locus of control articles (M-7.28); F(l,29)-13.18, p 
•0011) and more confident about judgments concerning empirical 
(K-7.74) than nonemp1r1ca1 articles (M-7.35; F(i,29)»10.26, p 
.0033). 

An Interaction between topic area and topic expertise 
Indicated that experts were more confident of judgments made about 
articles 1n their own area than their nonexpert area (for locus of 
control articles, locus of control experts M"7.47, hyperactivity 
experts M-7.07; for hyperactivity articles, hyperactivity experts 
H-7.95. locus of control experts N*7.67; F(l,29)-5.43, p .027). 

The amount of Information factor entered Into three 
significant effects. A main effect showed that participants were 
more confident about judgments based on more Information (keywords 
H*6.92, bibliographic N-7.34, abstract N-8.36; F(2,58)-35.72, p 
.0001), with the addition of an abstract precipitating a large 
Increase 1n confidence. An Interaction between amount of 
information and article content Indicated that the addition of 
Information had a greater Impact on the confidence In judgments 
made about empirical articles than nonemplrlcal ones 
(F(2,58)-4.5g, p .0142). 



Place Table 3 About Here 



Finally, an Interaction Involving amount of Information and 
topic area revealed that the effect of Information was greater on 
the confidence ratings for hyperactivity articles (keywords 
M«6.64, bibliographic M-7.34, abstract N-8.55) than for locus of 
control irtlcles (keywords K-7.18, bibliographic H»7.34, abstract 
K-8.18; F(2,58)»3.38. p .0407). 

Familiarity . The ANOVA performed on the measure of 
familiarity with the articles showed that participants were more 
familiar with hyperactivity articles (N-1.51) than locus of 
control articles (M-0.63;F(1,29)-13.81, p .0009) and with 
empirical articles (M-l.19) than nonemplrlcal ones (M-0.95; 
F(1.29)«12.06, p .0016). 
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The means underlying two significant interactions appear in 
Table 4. An Interaction between topic area and topic expertise 
revealed that participants were more familiar with articles in 
their own topic area (F(l,29)-46.59, p .0001). An Interaction 
involving topic area, topic expertise, and article content showed 
that experts were more likely to be familiar with empirical than 
nonemplrlcal articles In their own area, In their nonexpert area, 
however, no different existed between their familiarity with 
empirical and nonemplrlcal articles (F(l,29)"13.25, p .0011). 

Individual Difference Measures . 

ANOVAs with two between-subject factors (review publications 
and search experience) and two within-subject factors (article 
content and amount of Information) were conducted on participants* 
estimates that an article contained empirical data and their 
confidence In these judgments. 

The analysis of likelihood estimates revealed an Interaction 
between review publications, search experience, and article 
content (F(l,24)-4.97, p .0354). The Interaction Indicated that 
while all groups judged empirical articles more likely than 
nonemplrlcal articles to contain empirical data, participants with 
less than two published reviews and fewer than sixteen searches 
gave more similar estimates to empirical and nonemplrlcal articles 
than any other group. Table 5 presents the means underlying this 
effect. 



Place Table 5 About Here 



With regard to confidence 1n judgments, one two-way 
Interaction Involving review publications and article content 
proved significant. Participants with more than two published 
reviews showed a greater difference 1n their confidence about 
judgments of empirical and nonemplrlcal articles (empirical 
H-15.6, nonemplrlcal N-14.4) than did participants with less than 
two published reviews (empirical M-15.3, nonemplrical M-15.1; 
F(l,24)-4.45, p .0456), though not significantly so by the 
Newman-Keuls test. 

ANOVAs similar to those just described were conducted 
substituting the participants 1 cognitive complexity and tolerance 
of ambiguity for review publications and search experience. 

An Interaction between cognitive complexity and article 
content revealed that participants high in cognitive complexity 
showed a greater difference between their estimates that empirical 
and nonemplrlcal articles contained empirical data than did 
estimates by participants low in cognitive complexity 
(F(l,26)-6.28, p .0188), though not significantly so by the 
Newman-Keuls test. Table 6 presents the associated means. An 
Interaction between tolerance of ambiguity and article content 
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Indicated that participants high In tolerance of aablgulty showed 
a greater difference In their estimates that empirical and 
nonemplrlcal articles contained empirical data than did estimates 
by participants low In tolerance of aablgulty (F(l,26)"4.60, p 
.0415), though not significantly so by the Newman-Keuls test. 
Table 7 presents the underlying Means. The direction of effects 
Indicated that while all subjects estimated empirical articles 
were more likely to contain empirical data than nonemplrlcal 
articles, participants who were higher In cognitive complexity or 
tolerance of ambiguity tended to show this effect to a greater 
degree. 



Place Tables 6 and 7 About Here 



The analysis of the confidence 1n judgments revealed no 
significant effects Involving cognitive complexity or tolerance of 
ambiguity. 

Ancillary Measures . The ancillary measures were examined 
employing three sets of ANOVAs containing groupings of the 
between-subjects factors similar to those described above and the 
topic area as the sole withln-subjects variable* 

A topic area main effect Indicated that participants believed 
drug treatment of hyperactivity was a more important area to 
parents and the medical community (M-9.19) than locus of control 
and academic achievement was to psychologists and educators 
(H*8.06;F(1,29)7.46, p .0106). A topic expertise main effect 
revealed that locus of control experts believed that highly 
complex links Involving many variables were needed to explain the 
two relations (M-8.74) more than did hyperactivity experts 
(H-7.47;F(1.29)«4.52. p .0421). 

Three of the measures revealed significant topic area by 
topic expertise Interactions* These Indicated that (a) experts 
were more confident than nonexperts about their estimates of the 
size of the relation In their topic area (F(l,29)*31.68, p .0001); 
(b) experts were less likely than nonexperts to believe that the 
link between their topic area variables was complex and involved 
many mediators (F(l f 29)»8.59, p .0065), and (c) experts were less 
likely than nonexperts to agree that measurement problems rendered 
studies In their area Invalid (F(i,29)"15.80, p .0004). Table 8 
presents the means underlying these three interactions* 



Place Table 8 About Here 



Finally, an Interaction between tolerance of ambiguity and 
topic area Indicated that for locus of control articles 
participants high 1n tolerance of ambiguity had more confldenre In 
their estimates of relation strength (M"6.73) than those low in 
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tolerance of ambiguity (M-5.33), while for hyperactivity articles 
participants low In tolerance of ambiguity were more confident 
(M«6.47) than those high In tolerance for ambiguity 
(M«4.93;F(1,26}»4.55. p .0425). 

No effects Involving the review publications or search 
experience Individual differences proved significant. 

Discussion 

In the discussion that follows, constant reference will not 
be made to the fact that participants in this study were 
performing a specific type of literature review, namely an 
Integrative research review. This omission should not be 
construed to mean the results are assumed to be generallzable to 
other forms of review (see Cooper, 1985). The question of 
general 1z&b1 lit/ Is open for future research. 

Consistent with expectations, experts In a topic area were 
found to be more accurate judges than nonexperts of whether 
articles contained empirical data relevant to their topic. 
Unpredlcted, however* was the finding that nonexperts were as 
accurate as experts In judging empirical articles as relevant to 
their search. The greater overall accuracy of experts was due 
solely to their ability to judge nonemplrlcal articles as 
Irrelevant. 

One possiMe explanation for experts' greater ability to 
dismiss Irrelevant articles Is that they are more familiar with 
the irrelevant (In this case, nonemplrlcal but potentially 
related) literature. The measure of familiarity with the 
literature provided some qualified support for this 
notion— experts were more familiar than nonexperts with the 
nonemplrlcal literature in their area* Curiously, however, 
experts were also more familiar with the empirical literature, but 
this did not enhance their ability to Identify relevant articles. 

An explanation for this asymetry In the effect of familiarity 
might be that the information contained In the research literature 
databases gives clear clues that certain articles contain 
empirical data. When these clues are not present, the articles 
are. In fact, usually nonemplrlcal. Experts, due to familiarity, 
rule out certain of these Irrelevant articles. Nonexperts, on the 
other hand, are less likely to rule these articles out since such 
« judgment would be based solely on the fact that the available 
Information does not affirm the article as relevant. However, 1f 
this explanation were perfectly adequate, a statistical 
Interaction, Involving the topic area, expertise of the searcher, 
and the amount of available Information, would have been expected. 
THe "advantage" of expertise would be greatest when more 
Information was available about nonemplrlcal articles, that 1s, 
when articles were most recognizable and, therefore, the added 
familiarity of experts would have the most pronounced affect on 
judgments. 
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One Important Implication of the findings regarding expertise 
deserves special mention . As noted above, topic expertise had no 
impact on the searchers' ability to accurately detect articles 
that contained empirical data. Based on this, It might be 
concluded that experts are unlikely to become aware of more 
empirical references than nonexperts conducting similar searches. 
Such a conclusion Is not whole ly justified because experts often 
obtain Information about their topic through channels other than 
literature searches, including Invisible colleges and 
subscriptions to topic-related journals. It might be concluded, 
however, that If research reviews written by authors with previous 
expertise In an area contain more empirical references than 
reviews written by nonexperts this may be due to their access to 
other mechanisms for literature retrieval, rather than any 
advantage they have in using the research literature dats bases. 
Instead, the advantage of expertise in searching may manifest 
itself solely in how long it takes experts and nonexperts to 
conduct then task - experts may spend less time examining 
articles Irrelevant to their task. 

It would also seem Important tc ask why both locus of control 
and hyperactivity experts were more accurate at judging empirical 
articles on hyperactivity than on locus of control. This finding 
probably Involves the length and specificity of the information 
contained in the databases. In general, the number of keywords 
and the length of abstracts associated with hyperactivity articles 
were greater than those associated with locus of control articles. 
In addition, articles dealing with drug treatments of 
hyperactivity tended to deal only with this topic while locus of 
control and academic achievement articles often included tests of 
other hypotheses, most frequently other predictors of achievement. 

With regard to the amount of information available to 
searchers, it was somewhat surprising to find that keywords and 
bibliographic Information did not differ In their ability to 
elicit accurate judgments. While the title of an article contains 
an author-supplied list of keywords, the number of Important 
Identifiers In a title Is usually less than the number supplied by 
the database Indexers. Apparently, these add) clonal keywords were 
enough to offset any advantage bibliographic citations might have 
because they provide author and journal names, and the year of 
publication. 

Having an abstract In addition to the bibliographic 
Information was dearly an asset for accurately judging relevant 
articles but It was of no help for judging Irrelevant articles. 
Again, a possible key to understanding this finding Is that 
reference to erplrlcal data in abstracts makes judgments of most 
empirical articles relatively straightforward. The absence of 
reference to empirical data does not rule out Its presence In the 
article, thus leaving the searcher with some guesswork. This 
Interpretation is supported by the finding that participants 
expressed the greatest confidence In their judgments made about 
empirical articles based on abstract information. It is also 
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true, however, that participants expressed greater confidence In 
their judgments of nonempirlcal articles based on abstracts than 
on keywords or bibliographic Information. Yet, their accuracy for 
nonempirlcal articles based on abstracts was no better than that 
based on less Information, meaning this confidence was misplaced. 

This last finding withstanding. It Is still apparent that 
searchers display greater overall accuracy when basing judgments 
on abstracts rather than less Information. A second Implication 
of this study, then. Is that persons conducting manual searches of 
the research literature should not rule out studies based on 
keywords or bibliographic Information unless the article 1s 
clearly Irrelevant. Likewise, persons conducting 
computer-assisted searches should always request that abstracts 
accompany the computer's output. 

With regard to other Individual differences in search 
outcomes, greater experience with either the searching process or 
with publishing reviews was found to have a beneficial affect on 
judgment accuracy. As noted in the Introduction, both effects are 
probably attributable to greater experience with the searching 
process, which apparently facilitates a person's ability to 
predict which specific terms to look for when searching the 
literature. 

There was also some tentative evidence that persons high 1n 
cognitive complexity or tolerance of ambiguity, may be more 
accurate searchers. Though this result was not predicted (.« *■ its 
cause readily discernible. It is not inconsistent with what might 
be anticipated. Those high in tolerance for ambiguity are by 
definition (Frenkel-Brunswlk, 1954) better able to deal with 
indefinite and irregular information, while those high in 
cognitive complexity (Scott, 1962) have greater flexibility and 
tend to comfortably Incorporate and use discrepant (Incomplete, 
vague) Information in making judgments. Regardless of the precise 
process underlying this particular result, it is clear that one 
cannot overlook the "human" dimension of the .nterface between a 
searcher and the Information system, a process that may sometimes 
be viewed as purely mechanical. Apparently, persons confronted 
with the same task and information can obtain different search 
outcomes based on deeply-rooted differences in how they relate to 
their environments. 

finally, the results concerning general beliefs Indicated 
that experts saw their own topic areas as requiring fewer 
mediators and producing more valid results than nonexperts. Thus, 
fanl 1 iarl ty wl th a topic area did not produce a «anse of 
complexity and difficulty. Or perhaps, only those wno aie not 
discouraged by a topic's difficulties go on to publish research 
relevant to It. 

Based on the results of this study, then, future research 
might fruitfully be aimed at addressing the following questions: 
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(a) Is there something In *<M1t1on to familiarity that 
accounts for experts greater ability to accurately reject materia 1 
that is irrelevant to their search? 

(b) Why do keywords and bibliographic Information not differ 
in the accuracy of the judgments they elicit? 

(c) How do personality variables affect searching and what 
can be done to counteract detrimental personality effects on 
search outcomes? 

(d) Are the results of this study generallzable to literature 
reviews involving tasks other than research synthesis? 
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Table 1 

Estimates That an Article Contains Empirical Data as a Function of Topic. Topic Expertise, and 
Article Content 

Expertise 

Locus of Control Hyperactivity 

Topic Locus of Control Hyperactivity Locus of Control Hyperactivity 

Empirical 7.16a 7.80a 7.08a 7.99a 

Content 

Noncmpirical 4.83c 6.12b 5.86b 5.38c 

Note . Heans not sharing common subscript differ significantly by the Newman-Keuls test 
(P .05). 
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Table Z 

Estimates That an Article Contains Empirical Data as a Function of 
Information and Article Content 

Content 





Keywords 


7.05b 


5.54c 


Information 


Bibliographies 


7.17b 


5.47c 




Abstract 


6.26a 


5.63c 



Note. Means not sharing conmon subscript differ significantly by the 
-Keuls test (p .05). 
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Tab.e 3 

C onfidence in Judgment as a Function of Article Content and Information 



Content 
Empirical Honempirical 





Keywords 


7.02c 


6.82c 


Information 


Sibl iographics 


7.43bc 


7.25c 




Abstract 


8.75a 


7.98b 



Note . Means not sharing common subscript differ significantly by the 
Newman-Keuls test (p .05). 
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Table 4 

Familiarity With an Article as a Function of Topic. Topic Expertise and Article Content 



Expertise 

Locus of Control Hyperactivity 

Topic Locus of Control Hyperactivity locus of Control Hyperactivity 

Content Egpjrjcaj 1.28c 0.40c 0.14c 3.00a 

Noncapirlcal 0.87d 0.44c 0.17e 2.41b 

Note . Means not sharing cosnmon subscript differ significantly by the Newman-Keuls test (p .05). 
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Table 5 

Estimates That an Article Contains Empirical Data as a function of 
Published Reviews, Searching Experience, and Article Content 



Published Reviews 
<2 >2 



Searches 


<16 


>16 


<16 


>16 


Empirical 


7.06ab 


7.63a 


7.75a 


7.61a 


Nonempirical 


6. 26 be 


5.35c 


5.51c 


5.38c 



Note . Means not sharing common subscript differ significantly by the 
Newroan-Keuls test (p .05). 
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Table 6 

Estimates That an Article Contains Empirical Data as a Function of 
Cognitive Complexity and Article Content 

Cognitive Complexity 

Low High 

Empirical ^46 7.62 

Content 

Noncaplrical 5.83 5. IS 
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Table 7 

Estimates That an Article Contains Empirical Data as a Function of 
Tolerance of Ambiguity and Article Content 

Tolerance of Ambiguity 

Low High 

Empirica l ^41 7. 59 

Content 

Nonemplrical 5JJ3 SJA 
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Table 8 

General Beliefs About Topic Areas as a Function of Topic Expertise and Topic Area 

Expertise 

Locus of Control Hyperactivity 

Topic Locus of Control Hyperactivity Locut of Control Hyperactivity 

Confidence In 

Est1»tc* 7^00 L93 4^00 7_J2 



Complex Linkages 
Numerous Med 'a tors b 


2.38 


2.60 


3.12 


1.33 


Measurement Problem, 
Outcomes Invalid** 


-0.31 


0*87 


1.69 


1.20 



Notes , a* Higher scores mean greater confidence, 
b. Positive scores mean agreement. 



36 



